Systems and methods for interfacing with an occupant

ABSTRACT

Systems and methods communicate an intent of an autonomous vehicle externally. In one implementation, scan data of a field around a travel path of an autonomous vehicle is obtained. The scan data is captured using at least one sensor. An object in the field around the travel path is determined from the scan data. The object is determined to be mutable or immutable. A navigation condition associated with the object is determined based on whether the object is mutable or immutable. The navigation condition is correlated to a portion of the travel path. Control operation(s) of the autonomous vehicle is determined for the portion of the travel path in response to the navigation condition. A representation link between the control operation(s) of the autonomous vehicle and the object is generated. A representation of the field around the travel path is rendered and includes the representation link.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/455,279, entitled “Systems and Methods for Interfacing withan Occupant” and filed on Jun. 27, 2019, which claims benefit ofpriority to U.S. Provisional Patent Application No. 62/690,860, entitled“Method for Interfacing with an Occupant of an Autonomous Vehicle” andfiled on Jun. 27, 2018, each of which is specifically incorporated byreference herein in its entirety.

FIELD

Aspects of the present disclosure relate to systems and methods forinterfacing with an occupant and more particularly to a user interfaceproviding information relating to operation of an autonomous machine.

BACKGROUND

Autonomous machines transporting one or more occupants, includingwithout limitation, automobiles, terrestrial vehicles, aerial vehicles,aerospace vehicles, submersible vehicles, and/or the like, generallyoperate within their respective environment with limited or no input orcontrol by the occupants. However, the lack of operational engagementbetween the occupant and the autonomous machine may result in theoccupants lacking knowledge of future actions or an understanding of abasis of autonomous decisions taken by the autonomous machine. Such alack of knowledge and understanding may unnecessarily decrease aconfidence of the occupants in the operation of the autonomous machineor otherwise make the occupants uneasy. It is with these observations inmind, among others, that various aspects of the present disclosure wereconceived and developed.

SUMMARY

Implementations described and claimed herein address the foregoingproblems by providing systems and methods for communicating an intent ofan autonomous vehicle to an occupant of the autonomous vehicle. In oneimplementation, scan data of a field around a travel path along a routeof an autonomous vehicle is obtained. The scan data is captured using atleast one sensor. An object in the field around the travel path isdetermined from the scan data. The object is determined to be mutable orimmutable. A navigation condition associated with the object isdetermined based on whether the object is mutable or immutable. Thenavigation condition is correlated to a portion of the travel path.Control operation(s) of the autonomous vehicle is determined for theportion of the travel path in response to the navigation condition. Arepresentation link between the control operation(s) of the autonomousvehicle and the object is generated. A representation of the fieldaround the travel path is rendered. The representation includes therepresentation link and is presented to an interior of the autonomousvehicle using a presentation system.

Other implementations are also described and recited herein. Further,while multiple implementations are disclosed, still otherimplementations of the presently disclosed technology will becomeapparent to those skilled in the art from the following detaileddescription, which shows and describes illustrative implementations ofthe presently disclosed technology. As will be realized, the presentlydisclosed technology is capable of modifications in various aspects, allwithout departing from the spirit and scope of the presently disclosedtechnology. Accordingly, the drawings and detailed description are to beregarded as illustrative in nature and not limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example environment for communicating anintent of an autonomous vehicle to an occupant of the autonomousvehicle.

FIG. 2 is a diagram of an example interfacing system for interfacingwith an occupant of an autonomous vehicle.

FIG. 3 illustrates example operations for communicating an intent of anautonomous vehicle to an occupant of the autonomous vehicle.

FIG. 4 is a functional block diagram of an electronic device includingoperational units arranged to perform various operations of thepresently disclosed technology.

FIG. 5 is an example computing system that may implement various aspectsof the presently disclosed technology.

DETAILED DESCRIPTION

Aspects of the presently disclosed technology relate to systems andmethod for interfacing between an occupant and an autonomous machine,such as an autonomous vehicle. Generally, an interfacing systemgenerates a user interface for communicating an intent of the autonomousmachine to a user, such as an occupant of the autonomous machine. In oneaspect, scan data of a field around a travel path of the autonomousvehicle is continuously captured as an autonomous vehicle moves alongthe travel path. Object(s) in the field around the travel path aredetected from the scan data. Based on whether the object(s) aredetermined to be mutable or immutable, a navigation condition associatedwith each of the objects is determined. The navigation condition iscorrelated to a portion of the travel path, and a control operation(s)of the autonomous vehicle is determined in response to the navigationcondition. To communicate the intent of the autonomous vehicle foraddressing the object(s) in the field around the travel path, arepresentation link between the control operation(s) of the autonomousvehicle and each object is generated. The representation link mayinclude visual, audial, and/or tactile features communicating aconnection between the decisions and actions of the autonomous vehicleand the detection of the object(s). A representation of the field of thetravel path including the representation link is rendered and presentedto the occupant.

The various systems and methods disclosed herein generally provide forinterfacing between an occupant and an autonomous machine. The exampleimplementations discussed herein reference rendering a representation ofa field around a travel path of an autonomous vehicle to communicate anintent of an autonomous vehicle, including underlying decisions andactions, to occupants of the autonomous vehicle. However, it will beappreciated by those skilled in the art that the presently disclosedtechnology is application in other human machine interface contexts andto other manned and unmanned autonomous machines, including, withoutlimitation, terrestrial vehicles, aerial vehicles, aerospace vehicles,submersible vehicles, and/or the like. The presently disclosedtechnology may be further applicable to other types of machines and userdevices, such as a personal computer, workstation, mobile device, orother computing devices.

For a detailed description of an example environment 100 forcommunicating an intent of an autonomous vehicle 102 to an occupant ofthe autonomous vehicle 102 using an interfacing system 200, reference ismade to FIGS. 1-2. In one implementation, the autonomous vehicle 102 isautonomously navigating along a route and moving on a travel path 104along the route. The autonomous vehicle 102 is capable of operating tomove along the travel path 104 with limited input from occupants withinan interior of the autonomous vehicle 102. Stated differently, ratherthan the user having an operational engagement with the autonomousvehicle 102 to control its actions, the occupant may simply input adestination point or other instruction and the autonomous vehicle 102transports the occupant through a series of autonomous decisions. As aresult, the occupant may not have knowledge of actions planned or beingexecuted by the autonomous vehicle 102 or a basis of autonomousdecisions underlying those actions. For example, the occupant may notknow whether objects around the travel path 104 are detected by theautonomous vehicle 102 and/or that the autonomous vehicle 102 isresponding to the presence of such object properly.

To facilitate communication of the intent of the autonomous vehicle 102in such contexts, the interfacing system 200 of the autonomous vehicle102 includes a sensor system 106, a vehicle controller 202, and apresentation system 108. The sensor system 106 may be mounted on orotherwise deployed in the autonomous vehicle 102, and the presentationsystem 108 may be mounted or otherwise deployed in an interior of theautonomous vehicle 102 to communicate to the occupant(s). In someimplementations, the presentation system 108 may be part of a mobiledevice carried by the occupant in communication with the vehiclecontroller 202 for communicating to users, such as current, past, and/orfuture occupants.

In one implementation, the sensor system 106 scans a field 110 aroundthe travel path 104 of the autonomous vehicle 102. While the field 110is depicted in one area relative to the autonomous vehicle 102. It willbe appreciated that the field 110 may correspond to other areas,multiple areas, and/or a 360 degree area. One or more objects within thefield 110 that have the potential to impact the navigation of theautonomous vehicle 102 along the travel path 104 are detected based oncaptured scan data. For example, immutable objects, such as immutableobject 112, that are stationary within the field 110, and mutableobjects, such as mutable object 114, that are or may be moving withinthe field 110, are detected. More particularly, the sensor system 106includes at least one sensor configured to capture scan data of thefield 110. Object(s) in the field 110 around the travel path 104, suchas the immutable object 112 and the mutable object 114, are detectedfrom the scan data. The vehicle controller 202 determines whether theobjects 112 and 114 are immutable or mutable. In one implementation, thevehicle controller 202 determines whether the objects 112 and 114 aremutable or immutable based on an object type and a motion of the object.The object type may be determined based on a comparison of the object toa plurality of object profiles.

Based on whether the objects 112 and 114 are determined to be mutable orimmutable, the vehicle controller 202 determines a navigation conditionassociated with each of the objects 112 and 114. The navigationcondition generally corresponds to an impact of objects or an associatedaspect of the field 110 on the navigation of the autonomous vehicle 102.For example, the immutable object 112, such as a school zone sign, mayspecify a traffic regulation mandating a vehicle control parameterimpacting navigation of the autonomous vehicle 102, such as a reducedspeed within an associate school zone. As another example, the mutableobject 114 may have a trajectory that is estimated to intersect with thetravel path 104 of the autonomous vehicle 102, thereby impacting thenavigation of the autonomous vehicle 102. The navigation condition iscorrelated to a portion of the travel path 104, and one or more controloperations of the autonomous vehicle 102 are determined in response tothe navigation condition. In the example where the immutable object 112is a school zone sign, the vehicle controller 202 correlates the trafficregulations pertaining to school zones to the portion of the travel path104 within the school zone and determines control operation(s) forcomplying with the traffic regulations, such as braking or otherwisereducing speed. Similarly, the vehicle controller 202 correlates theportion of the travel path 104 at which the mutable object 114 isestimated to intersect with the autonomous vehicle 102 with the mutableobject 114 and determines control operation(s) of the autonomous vehicle102 for avoiding intersection with the mutable object 114.

To communicate the intent of the autonomous vehicle 102 for addressingthe object 112 and 114 in the field 110 around the travel path, thevehicle controller 202 generates a representation link between thecontrol operation(s) of the autonomous vehicle 102 and each object 112and 114. The representation link may include visual, audial, and/ortactile features communicating a connection between the decisions andactions of the autonomous vehicle 102 and the detection of the object112 and 114. For example, the representation link may include colorcoding, bounding boxes, highlighting, and/or other graphic or visualrepresentations linking the object 112 and 114 to an action of theautonomous vehicle 102. A representation of the field of the travel pathincluding the representation link is rendered, and the presentationsystem 108 presents the representation to the occupant within theautonomous vehicle 102. In one implementation, prior to or otherwise inconnection with communicating the representation link to the occupantusing the presentation system 108, the vehicle controller 202communicates with one or more subsystems 204 of the autonomous vehicle102 controlling various systems of the autonomous vehicle 102 toautonomously execute the control operation(s) of the autonomous vehicle102 for the portion of the travel path 104.

In some cases, the vehicle controller 202 may obtain navigation data,object profiles, scan data, traffic regulation data, known trafficregulations, weather data, and/or other raw or processed data over anetwork 206. In one implementation, the network 206 is used by one ormore computing or data storage devices, including one or more databases208, for providing or otherwise accessing relevant information fornavigating the autonomous vehicle, detecting objects in the field 110,determining navigation conditions, determining control operations forthe autonomous vehicle 102, generating representation links, and/orrendering representations of the field 110. The presentation system 108may be deployed in a mobile device and in communication with the vehiclecontroller 202 over the network 206. A server 210 may also host awebsite or an application that users visit to access information storedin the databases 208 and/or for accessing or interacting with theinterfacing system 200. The server 210 may be one single server, aplurality of servers with each such server being a physical server or avirtual machine, or a collection of both physical servers and virtualmachines. In another implementation, a cloud hosts one or more networkcomponents. The vehicle controller 202, the server 210, the sensorsystem 106, the presentation system 108, and other resources, such asthe database 208 or user devices, connected to the network 206 mayaccess one or more other servers or resources for access to one or morewebsites, information, web services interfaces, autonomous vehicles,and/or other services or information. The server 210 may also host asearch engine for accessing and modifying such information.

In one implementation, the autonomous vehicle 102 autonomously navigatesalong the travel path 104 according to a first set of navigationalparameters, while the sensor system 104 records a sequence of opticalscans of the field 110 around the autonomous vehicle 102. The sequenceof optical scans is scanned for immutable objects (e.g., traffic signs)and mutable objects, and the vehicle controller 202 renders arepresentation of the field 110 for presentation using the presentationsystem 108 based on the sequence of optical scans. For example, theimmutable object 112 may be a crosswalk sign, and the mutable object 114may be a pedestrian crossing in connection with the crosswalk sign.

In response to detecting the crosswalk sign and the pedestrian in thesequence of optical scans, the sign and the pedestrian may behighlighted in the representation of the field 110. In oneimplementation, the sign is validated to determine whether it iscurrently applicable to the navigation of the autonomous vehicle 102. Awarning window rendered in the representation presented with thepresentation system 108 (e.g., presented on an interior display) may bepopulated with an icon indicating that the sign is currently valid. Thevehicle controller 202 communicates with the vehicle subsystems 204 tomodify navigation of the autonomous vehicle according to trafficregulations regarding the crosswalk sign and the pedestrian. In responseto passing the sign, the icon may be removed from the warning windowrendered, and autonomous navigation may be resumed according to thefirst set of navigational parameters.

The autonomous vehicle 102 scans the field 110 around the autonomousvehicle 102 and detects immutable objects 112 (e.g., lane markers,traffic signs) in the field 110 that define navigational regulationsalong portions of the travel path 104 traversed by the autonomousvehicle 102. The autonomous vehicle 102 further detects any mutableobjects 114 (e.g., other vehicles, cyclists, pedestrians, and othermovable objects) that affect the route generation and navigationalactions of the autonomous vehicle. A representation of the field 110 maybe presented using the presentation system 108, for example on aninterior display mounted within an interior of the autonomous vehicle102 and visually accessible to an occupant. The mutable object(s) 114and the immutable objects 112 may be visually demarcated in therepresentation of the field 110 rendered on the display, withselectively rendered contextual prompts or notifications related tonavigational and other actions executed by the autonomous vehicle 102based on the mutable and immutable objects 112 and 114.

In one implementation, the autonomous vehicle 102 communicates itsinterpretation of its surrounding field 110—including both immutable andmutable objects 112 and 114 near or otherwise affecting the autonomousvehicle 102. The autonomous vehicle 102 further communicates therelevance of the objects 112 and 114 even if no longer in view and theeffects of the objects 112 and 114 on autonomous navigation of theautonomous vehicle 102. As such, the autonomous vehicle 102 increasesautonomous vehicle operation transparency, enabling the occupant toquickly verify that the autonomous vehicle is operating as expected andwith sufficient caution, thereby reducing occupant anxiety andincreasing trust in the autonomous operation of the autonomous vehicle102.

The interfacing system 200 may render the representation of the field110 in various manners. For example, a 2D or 3D representation of thefield 110 may be rendered and presented using the presentation system108, with the representation being populated with renderings of thedetected objects and representation links. The mutable object 114 may bedisplayed, for example, with bounding boxes around current and potentialfuture locations of the mutable object 114 based on the set oftrajectories. As described herein, the autonomous vehicle 102 may modifyor otherwise update its trajectory based on the mutable and immutableobjects (e.g., object 112 and 114) to avoid collision with a pedestrianor other vehicle while following speed and lane limitations defined bylocal speed limit signs and lane markers in the field. While executingthis trajectory, the autonomous vehicle 102 renders the updatedtrajectory on the interior display and annotates the trajectory and/orbounding boxes around other objects represented on the interior displayin order to visually communicate a link between control operationsexecuted by the autonomous vehicle 102 (e.g., braking, accelerating,turning, changing lanes) and the other detected objects.

In one implementation, the autonomous vehicle 102 can: detect anothervehicle or a pedestrian entering the road just ahead of the autonomousvehicle 102; render a bounding box around this vehicle or pedestrianrepresented on the interior display; calculate a new trajectory aroundthis vehicle or pedestrian; and render this new trajectory (and theprevious trajectory) on the interior display while executing this newtrajectory to avoid the vehicle or pedestrian, such as by rapidlyslowing ahead of or swerving around the vehicle or pedestrian. Thisrapid change in speed or direction may unsettle the occupant inside theautonomous vehicle 102 and prompt the user to look to the display forfeedback or information regarding this navigational action by theautonomous vehicle 102. By rendering such content on the interiordisplay in real-time, including the autonomous vehicle 102 perception ofthe field 110 and intended trajectory, the autonomous vehicle 102provides the occupant access to information in real time, which mayenhance the occupant experience and/or improve confidence in theautonomous vehicle 102.

In one implementation, while the autonomous vehicle 102 autonomouslytraverses a planned route, the autonomous vehicle 102 extracts trafficregulation data from the scan data that is associated with a detectedtraffic sign or similar immutable object. The vehicle controller 202interprets a regulation (e.g., a speed limit) or a warning (e.g.,slippery when wet, school zone, railroad crossing) represented by thistraffic sign, and highlights the traffic sign in a representation of thefield 110 rendered on the interior display of the presentation system108, for example. Alternatively or additionally, a traffic sign andwarning window rendered on the interior display may be populated with arepresentation (e.g., an icon) of the detected traffic sign. Theautonomous vehicle 102 may communicate context of the traffic sign tothe occupant via the interior display, including how the autonomousvehicle 102 is modifying its speed, lane, path planning, and/or othercontrol operation based on the traffic sign in the context of otherobjects detected in the field 110 around the autonomous vehicle 102.

For example, in response to identifying the immutable object 112 as aschool zone sign (or detecting an approaching known school zone), theautonomous vehicle 102 may: scan the field 110 around the autonomousvehicle 102 for children, pedestrians, or other mutable objects usingthe sensor system 106; reduce its speed responsive to detecting a childor pedestrian; visually indicate on the interior display of thepresentation system 108 that the autonomous vehicle 102 has detectedboth the school zone sign and the mutable object 114, which may beidentified as a child; and render a notification linking the reducedspeed of the autonomous vehicle 102 to the school zone sign and thechild on the interior display.

In another example, in response to determining that the immutable object112 is a yield sign, the autonomous vehicle 102 visually indicates thatthe autonomous vehicle 102 has detected the yield sign and scans thefield 110 around the autonomous vehicle 102 for another vehicleapproaching the autonomous vehicle 102. If another vehicle is notapproaching, the presentation system 108 may indicate that theautonomous vehicle 102 has right of way, clear the yield sign from therepresentation, and maintain its current operation. If another vehicleis approaching, the autonomous vehicle 102 may slow for the approachingvehicle and render a notification on the interior display linking thespeed reduction to the approaching vehicle and the yield sign.

The autonomous vehicle 102 generally provides a real-time representationof the perception of the autonomous vehicle 102 of the field 110,including the mutable and immutable objects 112 and 114 that may affectnavigation or operation of the autonomous vehicle. As the autonomousvehicle 102 autonomously navigates on the travel path 104 along theroute, the sensor system 106 continuously, regularly, intermittently, orupon prompt captures scan data of the field 110.

The sensor system 106 includes a suite of one or more sensors configuredto collect information about the environment of the autonomous vehicle102. In one implementation, the sensor system 106 includes a set of 360°LIDAR sensors arranged on the autonomous vehicle 102. For example, oneLIDAR sensor may be arranged at the front of the autonomous vehicle 102,and a second LIDAR sensor may be arranged at the rear of the autonomousvehicle 102. In another example, a cluster of LIDAR sensors may bearranged on the roof of the autonomous vehicle 102. Each LIDAR sensorcan output one three-dimensional distance map (or depth image)—such asin the form of a 3D point cloud representing distances between the LIDARsensor and external surface within the field of view of the LIDARsensor—per rotation of the LIDAR sensor (i.e., once per scan cycle). Thesensor system 106 may additionally or alternatively include: a set ofinfrared emitters configured to project structured light into the field110 near the autonomous vehicle 102; a set of infrared detectors (e.g.,infrared cameras); and/or the like.

In one implementation, the sensor system 106 includes one or more colorcameras facing outwardly from the front, rear, left lateral side, rightlateral, and/or other locations of the autonomous vehicle 102. Eachcamera can output a video feed containing a sequence of digitalphotographic images (or “frames”), such as at a rate of 20 Hz. Thesensor system 106 may alternatively or additionally include a set ofinfrared proximity sensors arranged along the perimeter of the base ofthe autonomous vehicle 102 and configured to output signalscorresponding to proximity of objects within one meter or other distanceof the autonomous vehicle 102.

The sensor system 106, the vehicle controller 202, and/or othercomputing units may fuse data streams from the LIDAR sensor(s), thecolor camera(s), the proximity sensor(s), and/or other sensors into oneoptical scan of the field 110 around the autonomous vehicle—such as inthe form of a 3D color map or 3D point cloud of roads, sidewalks,vehicles, pedestrians, and/or other features and objects in the field110 around the autonomous vehicle 102—per scan cycle. The sensor system102, the vehicle controller 202, or other computing unit may similarlytransform images output by the infrared detector(s) into a depth map ofthe field 110. In some implementations, the autonomous vehicle 102collects scan data and other data broadcast by other vehicles and/orstatic sensor systems over the network 206 and incorporates thecollected data into an optical scan to determine a state and context ofthe field 110 to elect subsequent control operations.

The interfacing system 200 may utilize a navigation map, localizationmap, and/or the like in determining control operations, navigation, andother autonomous decisions. The maps may be stored in local memory ofthe interfacing system 200 and/or acquired over the network 206. Thenavigation map defines a route for execution by the autonomous vehicle102, and the localization map is used to determine a location of theautonomous vehicle 102 in real space. In one implementation, the vehiclecontroller 202: determines the location of the autonomous vehicle 102 inreal space based on sensor data collected from the sensor system 106 andthe localization map; determines the context of the field 110 around theautonomous vehicle 102 based on these sensor data; determines a controloperation (e.g., a navigational decision) based on the context of thefield 110 around the autonomous vehicle 102, the real location of theautonomous vehicle (e.g., determined using the localization map), andthe navigation map. The vehicle controller 202 may implement a deeplearning and/or artificial intelligence model in generating thesedeterminations. The control operations are autonomously executed usingthe subsystems 204 within the vehicle (e.g., accelerator, brake, andsteering actuators). In one implementation, the vehicle controller 202compares features extracted from the optical scan to like featuresrepresented in the localization map to determine the geospatial locationand orientation of the autonomous vehicle 102 in real space. The vehiclecontroller 202 uses this information to elect a control operationaccordingly.

The interfacing system 200 may further implement a perception model,including, without limitation, integrated or discrete vehicle,pedestrian, traffic sign, traffic signal, and lane marker detectionmodels, to detect and identify mutable and immutable objects in thefield 110. The vehicle controller 202 may implement a navigation or pathplanning model (e.g., in the form of a convolutional neural network) toelect acceleration, braking, turning actions and/or other controloperations based on these mutable and immutable objects and the route ofthe autonomous vehicle 102.

It will be appreciated that the autonomous vehicle 102 can include anyother types of sensors and can implement any other scanning, signalprocessing, and autonomous navigation techniques or models to determineits geospatial position and orientation, to perceive objects in itsvicinity, and to elect control operations based on sensor data collectedthrough these sensors.

The presentation system 108 may be deployed in the interior of theautonomous vehicle 102, form part of a mobile device of an occupant inthe interior of the autonomous vehicle 102, and/or the like. Thepresentation system 108 may have various input and output devices. Inone implementation, the presentation system 108 includes an interiordisplay visible to an occupant within the passenger compartment of theautonomous vehicle 102. For example, the presentation system 108 mayinclude an LED-backlit LCD display or an OLED display arranged insidethe passenger compartment, such as on the dashboard or integrated into aseat headrest, for example.

During operation, in one implementation, the autonomous vehicle 102records scan data of the field 110 around the autonomous vehicle, suchas in the form of 3D LIDAR frames or 2D color images, and renders arepresentation in the form of a 2D projection of the scan data on theinterior display of the presentation system 108. The representation maybe a bird's-eye view of the autonomous vehicle 102 and surrounding field110. The autonomous vehicle 102 may be approximately centered on theinterior display. The representation may further include left, forward,and side passenger views rendered across the interior display. Theautonomous vehicle 102 may implement the perception model and thenavigation model to identify objects (e.g., the objects 112 and 114) inthe field 110, highlight or annotate representations of the objectsrendered on the interior display, regularly recalculate a trajectory ofthe autonomous vehicle, and to render the updated trajectory on theinterior display.

In one implementation, when rendering a representation of the field 110around the autonomous vehicle, the interfacing system 200 highlight themutable and/or immutable objects 112 and 114 affecting the navigation ofthe autonomous vehicle 102 or otherwise triggers the autonomous vehicleto execute control operations, such as a change in accelerator, brake,or steering position, using the representation link. For example, theinterfacing system 200 may render bounding boxes around other vehicles,pedestrian, cyclists, traffic cones, and other mutable objects near theautonomous vehicle 102 for depiction on the interior display. In anotherexample, the interfacing system 200 renders the field 110 around theautonomous vehicle generally in monochromatic or grayscale but rendersrelevant mutable objects in full color. The interfacing system 200 maysimilarly highlight traffic signs, traffic signals, lane markers, curbs,and other immutable objects near the autonomous vehicle 102 fordepiction on the interior display.

The interfacing system 200 may also render a traffic sign and warningwindow on the interior display—such as in a top-left corner of theinterior display—and populate this window with icons representingcurrent traffic signs, speed limits, and other regulations and guidancecurrent for the portion of the travel path 104 occupied by theautonomous vehicle 102, as described herein. As the autonomous vehicle102 detects new objects in the field 110, identifies new traffic signsor other immutable objects, and passes previously-detected objects androad signs while autonomously traversing a planned route, theinterfacing system 200 can regularly update the interior display of thepresentation system 108 to reflect a new set of valid traffic signs andobjects affecting the navigation and path planning.

In one implementation, the interfacing system 200 presents, updates,and/or animates content rendered on a fixed interior display of thepresentation system 108, while transporting an occupant from a pickuplocation to a drop-off location (e.g., in a rideshare context or othercontext). However, the interfacing system 200 can additionally oralternatively serve such content to a mobile computing device (e.g.,smartphone, tablet) of the occupant via the network 206 (e.g., directlyvia a local ad hoc wireless network or indirectly through a remotecomputer system). The mobile computing device can then render thecontent for consumption by the user, such as within a web browser ornative rideshare application.

As described herein, the interfacing system 200 may utilize alocalization map. The localization map may be annotated with: locationsof traffic signs; laws, rules, prompts, guidance, and other knowntraffic regulations represented by these traffic signs; and roadsegments in which the known traffic regulations apply. Throughoutoperation, the vehicle controller 202 queries the localization map forgeoreferenced traffic signs, zones, and/or other known trafficregulations for a portion of the travel path 104 that the autonomousvehicle 102 is currently occupying or approaching. The interfacingsystem 200 highlights an area of the field 110 rendered on the interiordisplay of the presentation system 108 with a location of an upcomingtraffic sign predicted by the localization map and renders an icon (orother textual or visual content) representative of this traffic sign onthe interior display (e.g., in the traffic sign and warning window),even if the traffic sign is not yet visible or even if its content isnot yet discernible by the autonomous vehicle 102. The autonomousvehicle can then execute methods and techniques described herein tohighlight the traffic sign, verify validity of the traffic sign, andautonomously navigate along the portion of the travel path 104 impactedaccording to the traffic regulations associated with the traffic sign.

However, if the autonomous vehicle 102 approaches and then passes ageoreferenced location of a traffic sign indicated in the localizationmap but fails to detect this traffic sign at or near its expectedlocation, the interfacing system 200 can: preserve a bounding box aroundthe expected location of the traffic sign in the representation of thefield 110 rendered on the interior display; render a type, navigationalregulation, guidance, or other related content from the expected trafficsign on the interior display; and render a notification that theautonomous vehicle 102 expected the traffic sign in this location andthat the autonomous vehicle 102 will continue to operate as though thetraffic sign were present. Accordingly, the interfacing system 200 canpopulate the traffic sign and warning window with a representation ofthe expected traffic sign. The interfacing system 200 can also send scandata recorded by the autonomous vehicle while passing the expectedlocation of the traffic sign to a remote computer system, where removalof the traffic sign is verified and the localization map is updated. Theremote computer system may serve localization map updates over thenetwork 206 to autonomous vehicles operating in the geographic regionaccordingly.

For example, the autonomous vehicle 102 can be loaded with alocalization map including a layer defining geolocations of stop signswithin a geographic region. If a recent automotive accident, inclementweather, or road construction resulted in failure or inadvertent removalof a stop sign within this geographic region and the autonomous vehicle102 is approaching the geospatial location of this failed or removedstop sign, the interfacing system 200 predicts a presence of the stopsign based on stop signs locations stored in the localization map. Theinterfacing system 200 visually indicates the location and regulation(i.e., “stop”) specified by the stop sign in the representation of thefield 110 rendered on the interior display of the presentation system108 as the autonomous vehicle 102 approaches this geospatial location.From scan data, the interfacing system 200 may determine that the stopsign is not present at the expected geospatial location and render anotification on the interior display that the autonomous vehicle 102nonetheless intends to stop at the expected geospatial location of thestop sign. The interfacing system 200 populates a traffic sign andwarning window rendered on the interior display with a representation ofa stop sign, and the autonomously vehicle 102 automatically slows andthen stops at the location of the stop sign, even though the stop signis not currently present before resuming navigation through thisintersection.

In another example, lane markers have been removed from the portion ofthe travel path 104 currently occupied by the autonomous vehicle 102,such as during recent construction. In this case, the interfacing system200 may project georeferenced lane markers represented in thelocalization map onto the field 110 around the autonomous vehicle 102;navigate between predicted locations of these lane markers; and populatethe representation of the field 110 rendered on the interior displaywith virtual lane markers at these predicted locations to visuallyindicate the understanding by the autonomous vehicle 102 of lanelocations and its intent to navigate within these lanes.

Generally, the autonomous vehicle 102 can leverage data stored in thelocalization map to continue to operate according to known traffic signsin a geographic region even if these traffic signs are no longer presentand to communicate its intent to do so to an occupant, which may reduceconfusion, frustration, and or anxiety and increase trust.

As an example of the presently disclosed technology in the context ofthe mutable object 114, the interfacing system 200 implements autonomousnavigation techniques to regularly scan the field 110 around theautonomous vehicle 102, to detect nearby mutable objects 114, and torecalculate its upcoming navigation (in the context of trafficregulations and guidance) based on the type, location, and motion ofthese nearby objects 114. In one implementation, the interfacing system200 projects the upcoming trajectory into the representation of thefield 110 rendered on the interior display of the presentation system108, such as in the form of a solid or dashed green line representingthe intended path of the autonomous vehicle 102 from its currentlocation. The interfacing system 200 may regularly repeat this processto rescan the nearby field, recalculate a trajectory, and update therepresentation presented with the presentation system 108 to reflect theupdated trajectory.

During the process of calculating the trajectory of the autonomousvehicle 102, the interfacing system 200 may: predict possibletrajectories (or sets of possible future locations) of other vehicle,pedestrians, cyclists, and/or other mutable objects nearby; recalculatethe autonomous vehicle trajectory based on possible trajectories orfuture locations of these other objects 114; and indicate these possibletrajectories or future locations of these other objects in therepresentation of the field 114 rendered using the presentation system108 to communicate to the occupant inside the autonomous vehicle why theautonomous vehicle has elected its current trajectory.

In one implementation, the autonomous vehicle: implements a perceptionmodel to identify a type of the mutable object 114 in the field 110(e.g., a pedestrian, a dog, a bicyclist, a motorcycle, a car, a truck,construction equipment); accesses a motion model for the type of object;leverages recent locations of the object 114 and the selected motionmodel to estimate a set of possible future locations and/or trajectoriesof the object 114, such as up to a limited time after the autonomousvehicle 102 is predicted to pass the object 114; define an avoidanceboundary around the object 114 and a set of possible future locations ofthe object 114; and then render an avoidance boundary (e.g., in the formof a bounding box) around the object 114 depicted in the representation,such as in addition to or instead of highlighting the object 114 itself.If the current trajectory of the autonomous vehicle 102 intersects theavoidance boundary, the autonomous vehicle 102 recalculates itstrajectory to avoid the avoidance boundary of the object 114 and updatesthe trajectory rendered with the presentation system 108 accordingly.

By rendering the avoidance boundary around the object 114 and therevised autonomous vehicle trajectory for avoiding the object 114 on theinterior display, the interfacing system 200 visually provides therepresentation link between the object 114 and the revised trajectorycurrently executed by the autonomous vehicle 102. In particular, therepresentation link may visually link a speed, lane, direction, or othercontrol operation currently executed by the autonomous vehicle 102according to the revised trajectory with avoidance of a collision with apossible future location of the object 114.

In one example, the interfacing system 200 detects a pedestrian who hasstepped into the road just ahead of the autonomous vehicle 102 andpredicts a possible future location of the pedestrian in the path of theautonomous vehicle 102 based on the pedestrian's current location and apedestrian motion model that predicts a high variability of speed anddirection of pedestrians. Accordingly, the interfacing system 200:calculates a revised trajectory; brakes rapidly according to thisrevised trajectory to avoid an avoidance boundary around the pedestrian;and simultaneously highlights the pedestrian, highlights the avoidanceboundary around the pedestrian (e.g., with a flashing red bounding box),and projects the updated trajectory in the representation of the field110 rendered on the interior display. The interfacing system 200 mayalso: render a representation link in the form of a notification linkingthe braking action to the pedestrian on the interior display, therebycommunicating the nature of the braking action in real-time to theoccupant.

In another example, the interfacing system 200 detects a vehicle turninginto the travel path 104 of the autonomous vehicle in a lane just aheadof the autonomous vehicle 102 and predicts a possible future location ofthis vehicle based on the vehicle's current location and a road vehiclemotion model that predicts a lower variability of speed and direction ofroad vehicles. Accordingly, the interfacing system 200: rapidly verifiesan open lane to the left side; calculates a revised trajectory to movethe autonomous vehicle 102 one lane to the left; rapidly veers left intothis lane according to this revised trajectory to avoid an avoidanceboundary around the vehicle; and simultaneously highlights the vehicle,highlights the avoidance boundary around the vehicle, and projects theupdated trajectory in the representation of the field rendered on theinterior display. The interfacing system 200 renders a representationlink correlating the braking action to the vehicle moving into thetravel path 104 on the interior display. For example, while navigatingaround the vehicle, the interfacing system 200 can render variousrepresentation links on the interior display: the original trajectory ofthe autonomous vehicle; a bounding box containing predicted locations ofthe vehicle as the autonomous vehicle approaches; the revised trajectoryof the autonomous vehicle 102 to avoid collision with the vehicle; and aprompt or notification highlighting a section of the revised trajectoryto avoid (e.g., swerve around) the vehicle. Furthermore, the interfacingsystem 200 can replay a sequence of optical images (e.g., a five-secondcolor video) recorded by the interfacing system 200 just before andduring the lane change to avoid the vehicle on the interior display,such as automatically or responsive to manual selection to review thisreplay from the user.

The interfacing system 200 can regularly repeat such steps for mutableobjects 114 to recalculate an avoidance boundary around the mutableobjects 114, revise the autonomous vehicle trajectory, and update therepresentation to reflect these changes, such as at a rate of 10 Hz, toprovide real-time feedback to the occupant regarding the intent andcontrol operations of the autonomous vehicle 102 in the context ofobject types, proximities, and trajectories near the autonomous vehicle102. Such steps may be concurrently or separately executed for multipleobjects in the field 110 near the autonomous vehicle 102.

For various immutable objects 112, the interfacing system 200 may takedifferent approaches depending on the nature of the immutable object 112in the field 110. For example, in the context of traffic signs, theinterfacing system 200 detects traffic signs in the field 110 near theautonomous vehicle 102 and to highlights the traffic signs in therepresentation of the field 110 rendered with the presentation system108, such as on the interior display. For example, in one implementationthe interfacing system 200 implements the perception model, thenavigation model, computer vision, and/or artificial intelligencetechniques to: detect the traffic sign in the field 110 around theautonomous vehicle 102; interpret traffic regulations, (e.g., navigationor path planning regulations, rules, laws, prompt, guidance, socialconstructions, etc.) from the traffic sign; link traffic regulation witha portion of the travel path 104 (e.g., particular geospatial location,such as a speed bump, or with a segment of road nearby, such as a mergelane following a yield traffic sign or a school zone following a schoolzone sign; modify the autonomous vehicle trajectory or execute anothercontrol operation at the portion of the travel path 104 accordingly; andupdate the representation to reflect the perception of the surroundingfield 110, the traffic sign, and the intended control operation executedby the autonomous vehicle 102 using one or more representation links.

In one implementation, once the interfacing system 200 detects thetraffic sign in the field 110 ahead of the autonomous vehicle 102, theinterfacing system 200: validate the traffic regulation corresponding tothe traffic sign based on other features or objects detected nearby;annotate or highlight the traffic sign and related objects in therepresentation of the field 110 rendered with the presentation system108; populate the traffic sign and warning window on the representationwith an icon representing the traffic sign; modify the speed, lane,navigation, or path planning of the autonomous vehicle 102 according tothe traffic sign (e.g., by slowing for a bump or for a school zone, byyielding to merging traffic, by changing lanes to avoid an upcomingconstruction zone); and simultaneously (or preemptively) presenting therepresentation link between the control operation and the traffic signin the representation.

In one particular example where the immutable object 112 is a speedlimit sign, the interfacing system 200 detects a speed limit signadjacent the segment of road currently occupied by the autonomousvehicle 102; highlights the speed limit sign in the representation ofthe field 110 rendered on the interior display; and populates thetraffic sign and warning window on the interior display with an iconrepresenting the speed limit read from the traffic sign. The autonomousvehicle 102 can preserve this speed limit icon in the traffic sign andwarning window on the interior display until the autonomous vehicle 102detects a speed limit sign indicating a different speed.

In another particular example, where the immutable object 112 is a speedbump sign, upon detecting the speed bump sign, the interfacing system200: predicts a location of a speed bump ahead based on the location ofthe traffic sign and/or detect the speed bump directly in scan datarecorded by the interfacing system 200 while approaching the speed bumpsign; highlights the speed bump in the representation of the fieldrendered on the interior display; populates the traffic sign and warningwindow on the interior display with an icon representing the speed bumpsign; and slows for the upcoming bump. To further link this controloperation (i.e., braking) to the speed bump sign, the the interfacingsystem 200 can also render a notification including “slowing forupcoming speed bump” on the interior display as the autonomous vehicle102 slows for the upcoming speed bump. The interfacing system 200further: determines that the autonomous vehicle 102 has passed the speedbump based on an output of an accelerator or IMU in the autonomousvehicle 102 or responsive to passing the speed bump sign; and removesthe icon representing the speed bump sign from the traffic sign andwarning window on the interior display accordingly.

In another particular example, where the immutable object 112 is aschool zone related sign, such as a sign stating “SCHOOL ZONE” or “25MPH WHEN CHILDREN PRESENT.” Upon detecting this school zone sign, theinterfacing system 200: scans the field 110 around the autonomousvehicle 102 for a pedestrian (e.g., any pedestrian, a pedestrian under athreshold height, etc.); highlights the sign in the representation ofthe field rendered on the interior display; and populates the sign andwarning window on the interior display with an icon representing theschool zone sign. If the interfacing system 200 does not detect apedestrian, the interfacing system 200 may: mute the school zone iconrendered on the interior display; render a notification including “nochildren present” on the interior display; and continue to navigate atits current speed.

On the other hand, if the interfacing system 200 does detect apedestrian nearby, the interfacing system 200: slows to the speedindicated on the school zone sign; highlights the pedestrian in therepresentation of the field 110 rendered on the interior display; andemphasizes the school zone icon rendered on the interior display orotherwise indicates that the restriction indicated by the school zonesign is currently valid on the interior display. The interfacing system200 may also render a notification that the autonomous vehicle 102 isslowing through the school zone on the interior display to link thespeed reduction to both the school zone sign and the pedestrian(s)detected nearby. For example, the interfacing system 200 may render—onthe interior display—“We're in a school zone, and I see children. I'mslowing down until we pass this school.”

Additionally or alternatively, the localization map stored in localmemory on the autonomous vehicle can include a layer indicatinggeoreferenced school zones within the geographic region, and theinterfacing system 200 may query the localization map for proximity to aschool zone. For example, upon confirming the proximity to a school zoneof the autonomous vehicle 102 based on the localization map, theinterfacing system 200 may: scan the field 110 around the autonomousvehicle 102 for a school zone sign; highlight the school zone sign inthe representation of the field 110 rendered on the interior display;populate the traffic sign and warning window rendered on the interiordisplay with a virtual icon indicating proximity to the school zone;scan the field 110 around the autonomous vehicle 102 for a pedestrian;indicate that the school zone sign and/or a regulation for active schoolzones is valid on the interior display responsive to detecting apedestrian; and then modify the speed, autonomous navigation, pathplanning, and/or other control operation, accordingly.

As the autonomous vehicle 102 navigates through the school zone atreduced speed, the interfacing system 200 continues to scan the field110 for an indicator that the autonomous vehicle 102 has passed theschool zone and/or that the navigational regulation linked to the schoolzone (e.g., indicated by the school zone sign) is no longer valid. Forexample, the interfacing system 200 may scan the field 110 around theautonomous vehicle 102 for a change in topography (e.g., presence ofhouses or retail structures facing the road rather than schoolbuildings) that may indicate that the autonomous vehicle 102 has exitedthe school zone. In another example, the interfacing system 200 can:continue to scan the field 110 around the autonomous vehicle 102 forpedestrians as the autonomous vehicle 102 passes through the schoolzone; and determine that the traffic sign is no longer valid if theautonomous vehicle 102 has moved more than a threshold distance (e.g.,50 meters) from a last detected pedestrian. Additionally oralternatively, the interfacing system 200 can query the localization mapfor a georeferenced boundary of the school zone and confirm that thetraffic regulation linked to the school zone is no longer valid for theautonomous vehicle 102 once the geospatial location of the autonomousvehicle 102 falls outside of the georeferenced boundary of the schoolzone. Once the interfacing system 200 thus determines that the trafficregulation linked to the school zone is no longer valid, such as due toabsence of pedestrians near the autonomous vehicle 102 or responsive toexit from the school zone, the autonomous vehicle 102 resumes navigationup to the speed limit posted on the road segment of the travel path 104.

In another particular example, the immutable object 112 defines aconstruction zone. The interfacing system 200 detects a sequence ofconstruction cones defining a construction zone and a construction zonesign that states “CONSTRUCTION ZONE 55 MPH.” Responsive to detectingthis construction sign, the interfacing system 200 scans the field 110around the construction zone for construction workers and movingconstruction equipment. If the interfacing system 200 does not detect aworker or moving construction equipment within the construction zone,the interfacing system 200: highlights the construction zone andconstruction sign within the representation of the field 110 rendered onthe interior display; populates the traffic sign and warning window onthe interior display with a muted icon representing the constructionzone; indicates that the construction sign is not currently valid due toabsence of construction workers on the interior display; and maintainsits current speed accordingly.

On the other hand, if the interfacing system 200 does detect aconstruction worker or moving construction equipment within or near theconstruction zone, the interfacing system 200: highlights theconstruction zones, construction workers, and/or construction equipmentin the field 110 rendered on the interior display; emphasizes theconstruction zone icon rendered in the traffic sign and warning windowon the interior display; and indicates that the autonomous vehicle 102has entered a construction zone on the interior display as theautonomous vehicle 102 slows to the posted speed within the constructionzone. The autonomous vehicle 102 can then resume the posted speed forthe road once the autonomous vehicle 102 passes the construction zone,for example determined by absence of construction cones in the field 110ahead of the autonomous vehicle 102.

In another particular example, the immutable object 112 defines a roadarea that may be hazardous under certain weather conditions. Forexample, the interfacing system 200 detects a sign stating “SLIPPERYWHEN WET”; detects moisture on the nearby road surface of the travelpath 104 or verifies a local weather condition within a weatherdatabase; selectively decreases the speed of the autonomous vehicle 102responsive to both presence of the traffic sign and moisture on the roadsurface; and indicates the traffic sign, the related road condition, andthe autonomous decision to reduce speed based on the traffic sign androad condition on the interior display.

Turning to FIG. 3, example operations 300 for communicating an intent ofan autonomous vehicle to an occupant of the autonomous vehicle areillustrated. In one implementation, an operation 302 obtains scan dataof a field around a travel path along a route of an autonomous vehicle.The scan data is captured using at least one sensor, which may be partof a sensory system deployed in the autonomous vehicle. An operation 304detects an object in the field around the travel path from the scandata, and an operation 306 determines whether the object is a mutableobject or an immutable object. In one implementation, the object isdetermined to be the mutable object or the immutable object based on anobject type and a motion of the object. The immutable object may be asign, a light, a traffic cone, a stationary traffic object, and/or thelike, and the mutable object may be an animal, a person, a vehicle, oranother moving object.

An operation 308 determines a navigation condition associated with theobject based on whether the object is the mutable object or theimmutable object. In one implementation, when the object is theimmutable object, the operation 308 determines the navigation conditionby extracting traffic regulation data from the scan data and correlatingthe traffic regulation data with a known traffic regulation. The trafficregulation data may include numbers, words, symbols, and/or the like.The known traffic regulation may be associated with at least one vehiclecontrol parameter. The vehicle control parameter(s) may include reducespeed, increase speed, stop, yield, merge, and/or the like.

The known traffic regulation may include, without limitation, one ormore traffic rules associated with a speed limit, a yield sign, a schoolzone, a road condition, a stop sign, a traffic light, a merge lane, aconstruction zone, a one way, a cross walk, and/or the like. The roadcondition includes, without limitation, a speed bump, a speed dip, aweather condition warning, a low clearance, and/or other conditions ofthe road or features of the travel route that will affect navigation oroperation of the autonomous vehicle.

In one implementation, when the object is the mutable object, theoperation 308 determines the navigation condition by determining a setpossible trajectories of the object and determining whether anytrajectory in the set of possible trajectories will intersect the travelpath of the autonomous vehicle. The set of possible trajectories may bedetermined by identifying an object type of the mutable object andidentifying a motion model for the object type. The set of possibletrajectories may be determined based on the motion model and one or morelocations of the object.

An operation 310 correlates the navigation condition to a portion of thetravel path. For example, the navigation condition may be correlated tothe portion of the travel path based on the set of possible trajectoriesfor the mutable object. The navigation condition may be correlated tothe portion of the travel path based on a localization map, the scandata, and/or other localization data. An operation 312 determines atleast one control operation of the autonomous vehicle for the portion ofthe travel path in response to the navigation condition. In oneimplementation, the control operation(s) are determined based on thevehicle control parameter(s) in a context of the portion of the travelpath. The control operation(s) is further based current weatherconditions when the road condition includes the weather conditionwarning.

An operation 314 generates a representation link between the controloperation(s) of the autonomous vehicle and the object, and an operation316 rendering a representation of the field around the travel path. Therepresentation includes the representation link between the controloperation(s) and the object. The representation link may include colorcoding, a bounding box, highlighting, and/or other visual, audial, ortactile features to link the control operation(s) to the object. Therepresentation is presented to an interior of the autonomous vehicleusing a presentation system, which may be mounted in or otherwisedeployed in the interior of the autonomous vehicle or be part of amobile device. Following or otherwise in connection with thepresentation of the representation, the control operation(s) of theautonomous vehicle may be autonomously executed for the portion of thetravel path. The representation link may be removed from therepresentation of the field following the autonomous execution of the atleast one control operation by the autonomous vehicle.

Turning to FIG. 4, an electronic device 400 including operational units402-412 arranged to perform various operations of the presentlydisclosed technology is shown. The operational units 402-412 of thedevice 400 are implemented by hardware or a combination of hardware andsoftware to carry out the principles of the present disclosure. It willbe understood by persons of skill in the art that the operational units402-412 described in FIG. 4 may be combined or separated into sub-blocksto implement the principles of the present disclosure. Therefore, thedescription herein supports any possible combination or separation orfurther definition of the operational units 402-412.

In one implementation, the electronic device 400 includes a display unit402 configured to display information, such as a graphical userinterface, and a processing unit 404 in communication with the displayunit 402 and an input unit 406 configured to receive data from one ormore input devices or systems. Various operations described herein maybe implemented by the processing unit 404 using data received by theinput unit 406 to output information for display using the display unit402.

Additionally, in one implementation, the electronic device 400 includesunits implementing the operations described with respect to FIG. 3. Forexample, the operations 308 and/or 310 may be implemented by anavigation condition determining unit 408, and the operations 312 and/or314 may be implemented by a link generation unit 410, which each mayinclude other units implementing each operation. The operation 316 maybe implemented by a rendering unit 412. In some implementations, acontrolling unit implements various operations for controlling theoperation of a vehicle based on the operations implemented by the otherunits.

Referring to FIG. 5, a detailed description of an example computingsystem 500 having one or more computing units that may implement varioussystems and methods discussed herein is provided. The computing system500 may be applicable to the measuring system 102 and other computing ornetwork devices. It will be appreciated that specific implementations ofthese devices may be of differing possible specific computingarchitectures not all of which are specifically discussed herein butwill be understood by those of ordinary skill in the art.

The computer system 500 may be a computing system is capable ofexecuting a computer program product to execute a computer process. Dataand program files may be input to the computer system 500, which readsthe files and executes the programs therein. Some of the elements of thecomputer system 500 are shown in FIG. 5, including one or more hardwareprocessors 502, one or more data storage devices 504, one or more memorydevices 506, and/or one or more ports 508-512. Additionally, otherelements that will be recognized by those skilled in the art may beincluded in the computing system 500 but are not explicitly depicted inFIG. 5 or discussed further herein. Various elements of the computersystem 500 may communicate with one another by way of one or morecommunication buses, point-to-point communication paths, or othercommunication means not explicitly depicted in FIG. 5.

The processor 502 may include, for example, a central processing unit(CPU), a microprocessor, a microcontroller, a digital signal processor(DSP), and/or one or more internal levels of cache. There may be one ormore processors 502, such that the processor 502 comprises a singlecentral-processing unit, or a plurality of processing units capable ofexecuting instructions and performing operations in parallel with eachother, commonly referred to as a parallel processing environment.

The computer system 500 may be a conventional computer, a distributedcomputer, or any other type of computer, such as one or more externalcomputers made available via a cloud computing architecture. Thepresently described technology is optionally implemented in softwarestored on the data stored device(s) 504, stored on the memory device(s)506, and/or communicated via one or more of the ports 508-512, therebytransforming the computer system 500 in FIG. 5 to a special purposemachine for implementing the operations described herein. Examples ofthe computer system 500 include personal computers, terminals,workstations, mobile phones, tablets, laptops, personal computers,multimedia consoles, gaming consoles, set top boxes, and the like.

The one or more data storage devices 504 may include any non-volatiledata storage device capable of storing data generated or employed withinthe computing system 500, such as computer executable instructions forperforming a computer process, which may include instructions of bothapplication programs and an operating system (OS) that manages thevarious components of the computing system 500. The data storage devices504 may include, without limitation, magnetic disk drives, optical diskdrives, solid state drives (SSDs), flash drives, and the like. The datastorage devices 504 may include removable data storage media,non-removable data storage media, and/or external storage devices madeavailable via a wired or wireless network architecture with suchcomputer program products, including one or more database managementproducts, web server products, application server products, and/or otheradditional software components. Examples of removable data storage mediainclude Compact Disc Read-Only Memory (CD-ROM), Digital Versatile DiscRead-Only Memory (DVD-ROM), magneto-optical disks, flash drives, and thelike. Examples of non-removable data storage media include internalmagnetic hard disks, SSDs, and the like. The one or more memory devices506 may include volatile memory (e.g., dynamic random access memory(DRAM), static random access memory (SRAM), etc.) and/or non-volatilememory (e.g., read-only memory (ROM), flash memory, etc.).

Computer program products containing mechanisms to effectuate thesystems and methods in accordance with the presently describedtechnology may reside in the data storage devices 504 and/or the memorydevices 506, which may be referred to as machine-readable media. It willbe appreciated that machine-readable media may include any tangiblenon-transitory medium that is capable of storing or encodinginstructions to perform any one or more of the operations of the presentdisclosure for execution by a machine or that is capable of storing orencoding data structures and/or modules utilized by or associated withsuch instructions. Machine-readable media may include a single medium ormultiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) that store the one or more executableinstructions or data structures.

In some implementations, the computer system 500 includes one or moreports, such as an input/output (I/O) port 508, a communication port 510,and a sub-systems port 512, for communicating with other computing,network, or vehicle devices. It will be appreciated that the ports508-512 may be combined or separate and that more or fewer ports may beincluded in the computer system 500.

The I/O port 508 may be connected to an I/O device, or other device, bywhich information is input to or output from the computing system 500.Such I/O devices may include, without limitation, one or more inputdevices, output devices, and/or environment transducer devices.

In one implementation, the input devices convert a human-generatedsignal, such as, human voice, physical movement, physical touch orpressure, and/or the like, into electrical signals as input data intothe computing system 500 via the I/O port 508. Similarly, the outputdevices may convert electrical signals received from computing system500 via the I/O port 508 into signals that may be sensed as output by ahuman, such as sound, light, and/or touch. The input device may be analphanumeric input device, including alphanumeric and other keys forcommunicating information and/or command selections to the processor 502via the I/O port 508. The input device may be another type of user inputdevice including, but not limited to: direction and selection controldevices, such as a mouse, a trackball, cursor direction keys, ajoystick, and/or a wheel; one or more sensors, such as a camera, amicrophone, a positional sensor, an orientation sensor, a gravitationalsensor, an inertial sensor, and/or an accelerometer; and/or atouch-sensitive display screen (“touchscreen”). The output devices mayinclude, without limitation, a display, a touchscreen, a speaker, atactile and/or haptic output device, and/or the like. In someimplementations, the input device and the output device may be the samedevice, for example, in the case of a touchscreen.

The environment transducer devices convert one form of energy or signalinto another for input into or output from the computing system 500 viathe I/O port 508. For example, an electrical signal generated within thecomputing system 500 may be converted to another type of signal, and/orvice-versa. In one implementation, the environment transducer devicessense characteristics or aspects of an environment local to or remotefrom the computing device 500, such as, light, sound, temperature,pressure, magnetic field, electric field, chemical properties, physicalmovement, orientation, acceleration, gravity, and/or the like. Further,the environment transducer devices may generate signals to impose someeffect on the environment either local to or remote from the examplecomputing device 500, such as, physical movement of some object (e.g., amechanical actuator), heating or cooling of a substance, adding achemical substance, and/or the like.

In one implementation, a communication port 510 is connected to anetwork by way of which the computer system 500 may receive network datauseful in executing the methods and systems set out herein as well astransmitting information and network configuration changes determinedthereby. Stated differently, the communication port 510 connects thecomputer system 500 to one or more communication interface devicesconfigured to transmit and/or receive information between the computingsystem 500 and other devices by way of one or more wired or wirelesscommunication networks or connections. Examples of such networks orconnections include, without limitation, Universal Serial Bus (USB),Ethernet, Wi-Fi, Bluetooth®, Near Field Communication (NFC), Long-TermEvolution (LTE), and so on. One or more such communication interfacedevices may be utilized via the communication port 510 to communicateone or more other machines, either directly over a point-to-pointcommunication path, over a wide area network (WAN) (e.g., the Internet),over a local area network (LAN), over a cellular (e.g., third generation(3G), fourth generation (4G) network, or fifth generation (5G)), or overanother communication means. Further, the communication port 510 maycommunicate with an antenna for electromagnetic signal transmissionand/or reception. In some examples, an antenna may be employed toreceive Global Positioning System (GPS) data to facilitate determinationof a location of a machine, vehicle, or another device.

The computer system 500 may include a sub-systems port 512 forcommunicating with one or more systems related to a vehicle to controlan operation of the vehicle and/or exchange information between thecomputer system 500 and one or more sub-systems of the vehicle. Examplesof such sub-systems of a vehicle, include, without limitation, imagingsystems, radar, lidar, motor controllers and systems, battery control,fuel cell or other energy storage systems or controls in the case ofsuch vehicles with hybrid or electric motor systems, autonomous orsemi-autonomous processors and controllers, steering systems, brakesystems, light systems, navigation systems, environment controls,entertainment systems, and the like.

In an example implementation, navigation condition information,representation information, and software and other modules and servicesmay be embodied by instructions stored on the data storage devices 504and/or the memory devices 506 and executed by the processor 502. Thecomputer system 500 may be integrated with or otherwise form part of avehicle. In some instances, the computer system 500 is a portable devicethat may be in communication and working in conjunction with varioussystems or sub-systems of a vehicle.

The present disclosure recognizes that the use of such information maybe used to the benefit of users. For example, the location informationof a vehicle may be used to provide targeted information concerning a“best” path or route to the vehicle and to avoid surface hazards.Accordingly, use of such information enables calculated control of anautonomous vehicle. Further, other uses for location information thatbenefit a user of the vehicle are also contemplated by the presentdisclosure.

Users can selectively block use of, or access to, personal data, such aslocation information. A system incorporating some or all of thetechnologies described herein can include hardware and/or software thatprevents or blocks access to such personal data. For example, the systemcan allow users to “opt in” or “opt out” of participation in thecollection of personal data or portions thereof. Also, users can selectnot to provide location information, or permit provision of generallocation information (e.g., a geographic region or zone), but notprecise location information.

Entities responsible for the collection, analysis, disclosure, transfer,storage, or other use of such personal data should comply withestablished privacy policies and/or practices. Such entities shouldsafeguard and secure access to such personal data and ensure that otherswith access to the personal data also comply. Such entities shouldimplement privacy policies and practices that meet or exceed industry orgovernmental requirements for maintaining the privacy and security ofpersonal data. For example, an entity should collect users' personaldata for legitimate and reasonable uses and not share or sell the dataoutside of those legitimate uses. Such collection should occur onlyafter receiving the users' informed consent. Furthermore, third partiescan evaluate these entities to certify their adherence to establishedprivacy policies and practices.

The system set forth in FIG. 5 is but one possible example of a computersystem that may employ or be configured in accordance with aspects ofthe present disclosure. It will be appreciated that other non-transitorytangible computer-readable storage media storing computer-executableinstructions for implementing the presently disclosed technology on acomputing system may be utilized.

In the present disclosure, the methods disclosed may be implemented assets of instructions or software readable by a device. Further, it isunderstood that the specific order or hierarchy of steps in the methodsdisclosed are instances of example approaches. Based upon designpreferences, it is understood that the specific order or hierarchy ofsteps in the method can be rearranged while remaining within thedisclosed subject matter. The accompanying method claims presentelements of the various steps in a sample order, and are not necessarilymeant to be limited to the specific order or hierarchy presented.

The described disclosure may be provided as a computer program product,or software, that may include a non-transitory machine-readable mediumhaving stored thereon instructions, which may be used to program acomputer system (or other electronic devices) to perform a processaccording to the present disclosure. A machine-readable medium includesany mechanism for storing information in a form (e.g., software,processing application) readable by a machine (e.g., a computer). Themachine-readable medium may include, but is not limited to, magneticstorage medium, optical storage medium; magneto-optical storage medium,read only memory (ROM); random access memory (RAM); erasableprogrammable memory (e.g., EPROM and EEPROM); flash memory; or othertypes of medium suitable for storing electronic instructions.

While the present disclosure has been described with reference tovarious implementations, it will be understood that theseimplementations are illustrative and that the scope of the presentdisclosure is not limited to them. Many variations, modifications,additions, and improvements are possible. More generally, embodiments inaccordance with the present disclosure have been described in thecontext of particular implementations. Functionality may be separated orcombined in blocks differently in various embodiments of the disclosureor described with different terminology. These and other variations,modifications, additions, and improvements may fall within the scope ofthe disclosure as defined in the claims that follow.

What is claimed is:
 1. A method of communicating an intent of a mobile device, the method comprising: obtaining scan data of a field around the mobile device, the scan data captured using at least one sensor; determining whether an object detected from the scan data in the field around the mobile device is stationary or moving; determining a navigation condition associated with the object based on whether the object is stationary or moving; correlating the navigation condition to a portion of a travel path of the mobile device; determining at least one control operation of the mobile device for the portion of the travel path in response to the navigation condition; generating a representation link between the at least one control operation and the object, the representation link presented using a presentation system; and autonomously executing the at least one control operation of the mobile device for the portion of the travel path.
 2. The method of claim 1, wherein the navigation condition is determined based on a known traffic regulation when the object is stationary and associated with traffic regulation data.
 3. The method of claim 2, wherein the traffic regulation data is extracted from the scan data and correlated with the known traffic regulation.
 4. The method of claim 2, wherein the known traffic regulation is associated with at least one control parameter and the at least one control operation is determined based on the at least one control parameter in a context of the portion of the travel path.
 5. The method of claim 1, wherein the at least one control operation is further determined based on current weather conditions correlated to the portion of the travel path.
 6. The method of claim 1, wherein the navigation condition is determined based on a set of possible trajectories of the object when the object is moving.
 7. The method of claim 6, wherein the navigation condition is determined according to whether any of the set of possible trajectories will insert the travel path.
 8. The method of claim 6, wherein the set of possible trajectories is determined using a motion model for an object type of the object and one or more locations of the object in the field.
 9. One or more tangible non-transitory computer-readable storage media storing computer-executable instructions for performing a computer process on a computing system, the computer process comprising: obtaining scan data of a field around a mobile device, the scan data captured using at least one sensor; determining whether an object detected from the scan data in the field around the mobile device is stationary or moving; determining a navigation condition associated with the object based on whether the object is stationary or moving; correlating the navigation condition to a portion of a travel path of the mobile device; determining at least one control operation of the mobile device for the portion of the travel path in response to the navigation condition; generating a representation link between the at least one control operation and the object; and outputting the representation link between the at least one control operation and the object, the representation link presented using a presentation system.
 10. The one or more tangible non-transitory computer-readable storage media of claim 9, wherein the navigation condition is determined based on a known traffic regulation when the object is stationary and associated with traffic regulation data.
 11. The one or more tangible non-transitory computer-readable storage media of claim 10, wherein the traffic regulation data is extracted from the scan data and correlated with the known traffic regulation.
 12. The one or more tangible non-transitory computer-readable storage media of claim 10, wherein the known traffic regulation is associated with at least one control parameter and the at least one control operation is determined based on the at least one control parameter in a context of the portion of the travel path.
 13. The one or more tangible non-transitory computer-readable storage media of claim 9, wherein the navigation condition is determined based on a set of possible trajectories of the object when the object is moving.
 14. The one or more tangible non-transitory computer-readable storage media of claim 13, wherein the navigation condition is determined according to whether any of the set of possible trajectories will insert the travel path.
 15. The one or more tangible non-transitory computer-readable storage media of claim 13, wherein the set of possible trajectories is determined using a motion model for an object type of the object and one or more locations of the object in the field.
 16. A system of communicating an intent of a mobile device, the system comprising: a sensor system deployed in the mobile device, the sensor system having at least one sensor capturing scan data of a field around a travel path of the mobile device; a controller in communication with the sensor system, the controller configured to determine a navigation condition associated with an object detected in the field around the travel path using the scan data, the navigation condition determined based on whether the object is stationary or moving and correlated to a portion of the travel path, the controller configured to determine at least one control operation of the mobile device for the portion of the travel path in response to the navigation condition; and a presentation system in communication with the controller, the presentation system configured to present a representation link between the at least one control operation and the object.
 17. The system of claim 16, wherein the navigation condition is determined based on a known traffic regulation when the object is stationary and associated with traffic regulation data.
 18. The system of claim 17, wherein the known traffic regulation is associated with at least one control parameter and the at least one control operation is determined based on the at least one control parameter in a context of the portion of the travel path.
 19. The system of claim 16, wherein the navigation condition is determined based on a set of possible trajectories of the object when the object is moving, the navigation condition determined according to whether any of the set of possible trajectories will insert the travel path.
 20. The system of claim 19, wherein the set of possible trajectories is determined using a motion model for an object type of the object and one or more locations of the object in the field. 